Skip to content

Fix retry bug in tree model query which ignore the time filter#18179

Merged
JackieTien97 merged 4 commits into
masterfrom
fix-retry-bug
Jul 14, 2026
Merged

Fix retry bug in tree model query which ignore the time filter#18179
JackieTien97 merged 4 commits into
masterfrom
fix-retry-bug

Conversation

@JackieTien97

@JackieTien97 JackieTien97 commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Description

Problem

The first query dispatch may fail for a retryable reason. In the reported case, the DataNode hosting the region leader had reached the upper limit of 1,000 internal connections.

QueryExecution then analyzes the same parsed statement again. However, tree-model analysis previously modified parser-owned AST state in place, including extracting the global time predicate from WHERE. As a result, the retry could analyze a statement different from the original SQL and lose its time filter. Other analysis-time mutations, such as normalized expressions, ORDER BY, LIMIT/OFFSET pushdown, and template-specific rewrites, had the same retry-safety risk.

Changes

  • Make global-time-predicate extraction and predicate simplification non-mutating. They now return the extracted time predicate and a residual predicate, path-copying only expression branches that need to change.
  • Add a per-query TreeAnalysisMutationJournal for tree-model analysis:
    • lazily record only parser-owned fields that are actually replaced;
    • use copy-on-write for nested mutable state such as ORDER BY;
    • avoid an unconditional deep copy of the complete tree-query AST.
  • Define the analysis-attempt lifecycle in QueryExecution:
    • begin an attempt before analysis;
    • keep its working state while generated plans and the scheduler still use it;
    • after a retryable dispatch failure, stop and clean up the failed attempt before rolling it back;
    • reanalyze from the original parser-owned state;
    • commit only after the query reaches a successful running or finished state, and roll back on analysis or terminal failures.
  • Keep implicit DEVICE and TIME sort keys for DeviceView in planning-owned state instead of appending them to the parsed QueryStatement.
  • Analyze parser-owned table-model SHOW DEVICES and COUNT DEVICES statements on per-analysis working copies. Internal schema-fetch ShowDevice statements retain their pre-populated traversal state because that state is request input rather than derived analysis data.
  • Intentionally share raw WHERE expressions in table-model working copies. This analysis path treats them as read-only, replaces rewritten roots only on the working statement, and avoids a deep-copy cost on normal one-attempt queries.

These changes ensure that every retry observes SQL semantics equivalent to the original statement while limiting copying to state that analysis actually needs to modify.

Tests

Added regression unit tests covering:

  • repeated analysis preserving tree-query time filters;
  • mixed AND, OR, and NOT predicates without mutating input expressions;
  • retry safety for GROUP BY, HAVING, ORDER BY, SELECT INTO, LIMIT/OFFSET, count_time(*), and template analysis;
  • analysis-attempt commit and rollback behavior for successful retries and exhausted retry limits;
  • two-stage DeviceView and SortNode planning without modifying or accumulating sort keys in the parsed statement;
  • repeated analysis of table-model SHOW DEVICES and COUNT DEVICES;
  • preservation of pre-populated internal ShowDevice traversal state.

@codecov

codecov Bot commented Jul 11, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 80.50139% with 70 lines in your changes missing coverage. Please review.
✅ Project coverage is 42.60%. Comparing base (c716365) to head (e77abee).
⚠️ Report is 14 commits behind head on master.

Files with missing lines Patch % Lines
.../db/queryengine/plan/execution/QueryExecution.java 61.29% 12 Missing ⚠️
...gine/plan/analyze/TreeAnalysisMutationJournal.java 89.90% 11 Missing ⚠️
...gine/plan/analyze/TemplatedAggregationAnalyze.java 64.00% 9 Missing ⚠️
...ression/visitor/predicate/PredicateSimplifier.java 70.96% 9 Missing ⚠️
...db/db/queryengine/plan/analyze/PredicateUtils.java 85.45% 8 Missing ⚠️
.../db/queryengine/plan/planner/TreeModelPlanner.java 0.00% 8 Missing ⚠️
...db/db/queryengine/plan/analyze/AnalyzeVisitor.java 82.85% 6 Missing ⚠️
...he/iotdb/db/queryengine/plan/planner/IPlanner.java 0.00% 3 Missing ⚠️
.../db/queryengine/plan/analyze/TemplatedAnalyze.java 0.00% 2 Missing ⚠️
...ne/plan/relational/analyzer/StatementAnalyzer.java 80.00% 2 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #18179      +/-   ##
============================================
+ Coverage     42.04%   42.60%   +0.56%     
  Complexity      318      318              
============================================
  Files          5312     5323      +11     
  Lines        375102   376815    +1713     
  Branches      48422    48742     +320     
============================================
+ Hits         157693   160548    +2855     
+ Misses       217409   216267    -1142     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@sonarqubecloud

Copy link
Copy Markdown

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a retry-safety bug in tree-model query analysis where analysis-time in-place mutations of the parser-owned AST could cause subsequent retry attempts to observe different semantics (notably losing global time filters). It introduces copy-on-write / journaling for tree-model analysis mutations, adjusts planner/execution lifecycle to explicitly begin/commit/rollback analysis attempts, and adds regression coverage for both tree and table model re-analysis.

Changes:

  • Add a tree-model TreeAnalysisMutationJournal and use it to make analysis-time mutations retry-safe (copy-on-write for nested mutable state; non-mutating time predicate extraction + predicate simplification).
  • Define an analysis-attempt lifecycle in QueryExecution and connect it to planners via new IPlanner attempt hooks.
  • Add unit tests covering repeated analysis, dispatch-retry behavior, and non-mutation guarantees across ORDER BY, LIMIT/OFFSET pushdown, templates, and table-model device queries.

Reviewed changes

Copilot reviewed 26 out of 26 changed files in this pull request and generated no comments.

Show a summary per file
File Description
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/queryengine/plan/statement/QueryStatementTest.java Adds ORDER BY copy-on-write regression test for raw vs normalized expression handling.
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/queryengine/plan/relational/analyzer/AnalyzerTest.java Adds table-model SHOW/COUNT DEVICES re-analysis tests and internal ShowDevice traversal-state preservation test.
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/queryengine/plan/planner/LogicalPlanBuilderTest.java Verifies DeviceView planning doesn’t mutate parsed sort items and uses implicit merge keys correctly.
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/queryengine/plan/execution/QueryExecutionRetryTest.java Adds tests for analysis-attempt commit/rollback sequencing across retry success/exhaustion.
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/queryengine/plan/analyze/TemplatedAnalyzeRetryTest.java Adds template-analysis retry-safety tests (count_time(*) and LIMIT/OFFSET pushdown).
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/queryengine/plan/analyze/PredicateUtilsTest.java Adds non-mutating time-predicate extraction and copy-on-write predicate simplification tests.
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/queryengine/plan/analyze/AnalyzeTest.java Adds broad regression coverage ensuring repeated analysis preserves semantics across many tree-model features.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/statement/component/WhereCondition.java Adds factory to wrap already-normalized predicates without rebuilding AST.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/statement/component/OrderByComponent.java Adds copyOf to enable analysis-time copy-on-write updates while sharing immutable expression nodes.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/statement/component/GroupByComponent.java Adds analyze-only setter to avoid renormalizing already-normalized expressions during rollback/restore.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/sql/ast/ShowDevice.java Implements copyForAnalysis() to isolate SQL-parsed ShowDevice from analyzer mutations.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/sql/ast/CountDevice.java Implements copyForAnalysis() to isolate SQL-parsed CountDevice from analyzer mutations.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/sql/ast/AbstractQueryDeviceWithCache.java Adds “parsed-input-only” copy constructor for per-analysis working statements (shares raw WHERE intentionally).
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/analyzer/StatementAnalyzer.java Uses working copies for SQL device queries while preserving internal ShowDevice prepared traversal state.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/TreeModelPlanner.java Wires planner attempt lifecycle to TreeAnalysisMutationJournal.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/LogicalPlanBuilder.java Avoids mutating parsed QueryStatement sort keys for DeviceView implicit merge keys; stores merge order parameter in Analysis.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/IPlanner.java Adds default analysis-attempt lifecycle hooks (begin/rollback/commit).
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/expression/visitor/predicate/PredicateSimplifier.java Converts predicate simplification to non-mutating, reconstructing only when needed (copy-on-write).
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/execution/QueryExecution.java Adds explicit attempt begin/commit/rollback behavior aligned with dispatch retry lifecycle.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/analyze/TreeAnalysisMutationJournal.java New journal implementing attempt-scoped undo/CoW for parser-owned tree-model AST mutations.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/analyze/TemplatedAnalyze.java Routes analysis-time ORDER BY updates through the mutation journal.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/analyze/TemplatedAggregationAnalyze.java Avoids in-place mutation of count_time(*) and journals LIMIT/OFFSET pushdown after eligibility checks.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/analyze/PredicateUtils.java Reworks global time predicate extraction to be non-mutating and returns residual predicate + value-filter flag.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/analyze/ConcatPathRewriter.java Routes SELECT/GROUP BY/ORDER BY analysis-time rewrites through mutation journal / CoW.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/analyze/AnalyzeVisitor.java Integrates journal into semantic-check-derived flags, time filter extraction, ORDER BY updates, and template path.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/analyze/Analyzer.java Ensures per-analyze attempt preparation and rollback-on-exception when Analyzer is invoked directly.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@JackieTien97 JackieTien97 merged commit e75d146 into master Jul 14, 2026
45 checks passed
@JackieTien97 JackieTien97 deleted the fix-retry-bug branch July 14, 2026 06:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants